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Software  Engineering  Institute 


S ome  C  ommon  M  i  scon cept i  on  s 

Process  maturity  Levels  2-3  focus  on  process  definition. 
Measurement  is  largely  postponed  until  Levels  4  and  5. 

Technical  Working  Groups  (TWGs)  should  be  organized  by 
CMM/CMMI  process  areas  -  therefore  there  should  be  a 
separate  “Measurement  TWG.” 

There  should  be  two  plans  -  one  that  guides  process  change 
and  one  that  guides  how  measurement  is  introduced  and 
implemented. 


Only  certain  specialized  roles 
require  competency  in 
measurement  skills. 
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Measurement  Activities  in  Process 
I  mprovement 

Measurement  is  an  essential  construct  of  process  - 
measurement  must  be  woven  into  the  fabric  of  process. 

Definition  of  measurement  should  be  an  essential  part  of  any 
process  definition  work. 

Measurement  focuses  continuous  process  improvement  -  it 
should  never  be  postponed  (although  the  sophistication  of  data 
collection  and  storage  mechanisms  should  evolve). 

Everyone  needs  to  understand  how 
to  effectively  monitor  and  improve 
their  work  processes  -  measurement  is  the  key. 
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Measurement  Pinpoints  Future 
I  mprovement  Opportunities 

It  has  generally  been  found  that  variation  in  a  few  of  the 
process  attributes  has  significant  impact  on  overall 
performance.  These  significant-few  process  attributes  are 
"key"  process  characteristics. 


These  key  process  characteristics  are  the  ones  we  should  be 
measuring. 


Nonconformance  Category 
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The  I DEALSM 
Approach 


Learning 


Stimulus  for  Change 


Acting 


Diagnosing 


SM  IDEAL  is  a  service  mark  of 
Carnegie  Mellon  University 


Establishing 
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The  I  initiating 
Phase 


Learning 


Stimulus  for  Change 


Diagnosing 


Acting 


Establishing 
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Key  Success  Factors  for  Effective  Launch 

The  Must  Haves: 

:  1 1  A  compelling  reason  for  change 

2  Leadership  of  the  change  effort  by  the  top  executive  in 
the  organization  —  responsibility  cannot  be  delegated 

3  Informed  commitment  of  the  top  management  team 

Designation  of  a  primary  change  agent  (the  EPG 
Leader)  and  an  adequate  mandate  for  change 

5  Sound  performance  measures  that  drive  change 
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5  P  erf  or  ma  n  ce  M  easu  r  es 
that  Drive  Change 


Measures  are  integral  to  effective  change.  A  goal-focused 
measurement  system  is  the  best  vehicle  for  galvanizing  management 
action  and  institutionalizing  the  targeted  changes. 

If  you  deploy  a  poor  set  of  measures  (or  none  at  all),  any  positive 
accomplishment  will  be  undermined  by  inefficiencies  as  managers 
chase  an  inconsistent  and/or  conflicting  set  of  targets. 

Performance  measures  must  be: 

•  Relevant.  Does  the  measure  have  a  significant,  demonstrable 
relation  to  strategy  and  business  goals? 

•  Reliable.  Will  the  measure  identify  strengths  or  weaknesses 
of  one  or  more  business  processes? 
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Performance  Measures: 

Reestablishing  Your  Business  Goals 

Performance  measures  must  be  tied  to  and  reflect  the  new  vision  that 
is  driving  the  change.  And  the  right  measures  must  be  forged  on 
business  goals  that  are  both  real  and  inspirational. 

For  most  organizations,  development  of  performance  measures  must 
be  preceded  by  reestablishing  or  refreshing  the  organization’s  values, 
vision,  mission  and  goals. 

When  these  are  undefined,  are  not  communicated,  or  are  not  inspiring 
to  people,  it  can  undermine  any  effort  to  improve. 

If  the  values,  vision,  mission  and  goals  are  poorly  defined,  then  work 
will  be  needed  before  performance  measures  can  be  considered — or 
before  other  positive  changes  can  be  achieved. 
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Project  Measures  Should  Ultimately 
Tie  to  Customer  Satisfaction 
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Learning 


Canugfe  Melon 

Software  Engineering  hstitute 


Assessing  Your 
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Establishing 


© — @ — ©- 

Organize  Conduct  Establish 

and  Prepare  Organizational  Technical 

Scan  VUxking 

Groups 
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Process  Change 
Methodology 


Acting  Learning 


© — © — © — © — ® 


Understand 
Project's 
Current  State 


Redesign 
the  Process 


Develop 

Solution 


Conduct 
Pilots)  and 
Evaluate 


Facilitate 

Organizational 

Learning 


Gem^Miost 


The  Process  Change  Method 


©  Organize  and  Prepare 
©  Conduct  Organizational  Scan 
®  Establish  Technical  Working  Groups 
©  Understand  Project’s  Current  State 
©  Redesign  the  Process 
©  Develop  Solution 
©  Conduct  Pilot(s)  and  Evaluate 
®  Facilitate  Organizational  Learning 
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PCM  1:  Organize  and  Prepare  . 
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When  Discussion  T urns  to  Problem- 


is  at  the  core  of  the  SEPG’s  activities. 


A  problem  is  simply  a  gap  between  what  is  desired  and  what 
exists. 


A  group  with  an  effective  problem-solving  process  meets  two 
conditions: 


1  members  use  a  systematic  process  for  solving  problems 


2  all  members  focus  on  the  same  step  of  the  problem¬ 
solving  process  at  the  same  time 


Adapted  from  [Senge  90,  pp.  234-235] 


©  2001  by  Carnegie  Mellon  University 


15 


GempMAost 


Generic  Problem-Solving  Process 


Define  the  problem 

Establish  criteria  for  evaluating  solutions 
Identify  root  causes 


Generate  alternative  solutions 


Evaluate  alternative  solutions 


Select  the  best  solution 


Develop  an  action  plan 
Implement  the  action  plan 

Evaluate  outcomes  and  the  process 


Adapted  from  [Schwarz  94,  p.  1 59] 
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9  E  val  uate  Outcomes  of  the  P  rocess 


Evaluation  through  measurement  is  the  step  most  often 
underemphasized. 

However,  evaluation  is  essential  for  a  group  that  values  valid 
information. 

When  groups  resist  evaluation,  several  interventions  are  possible: 

•  help  the  group  explore  its  reluctance  to  evaluate 

•  once  barriers  are  identified,  help  group  reframe  the  meaning  of 
evaluation  from  threat  to  one  in  which  members  seek 
continuous  improvement  of  their  work 

•  help  the  group  consider  how  comprehensive  their  evaluation 
needs  to  be 
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PCM  2:  Conduct  Organizational  Scan 
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PCM  2:  Conduct  Organizational  Scan 
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PCM  2:  Conduct  Organizational  Scan 


tii .  {  hi 
.  II 

.  ||iii  iii. 

iii  .11  i>i 
II V  i>i  II 


. . 

:  *.  *  a.1!'  * 

•  HIM  .11  Hill.11. 

1 !  H til  iii  If  in 
*.  'I . s . 


*•••••■•* 


iii  S  iii 

III."  .  II 
.11  III  III  . 
Ill  .Hill 

inn  'I'  H 


•  iii  in 

'S',:,  H  ",§,11 
iii  iii  ill  ijji  in 
ill  II  II  'll  H  II 


in. 

*'.T  * 

ii»  « 


. . . 

.  • 

•  •  III 

.11"  Ia"1! 

•  T  11 .  T*" 

•  H  III  "  ||  * 


iii  S  '*' 
in  •  .  II 
.11  iii  iii  . 

ill.  II  .Dili 
ll'ii  i»  H 


in  ■■■ 

"*.'*T*4 

'■*  «,i,  iif 
'S'  11 .  "li,11 

II  III  11  II 


'i'.  h  m 
in  .  .  II 

.11  in  iii  . 
ill.  II  .Him 
lliii  iii  II 


iii* 'll1  iii 


rr.ii  .i.1: 

iii  .lliii 
inn  iii  II 


•  •  •  •  • 


iii. 

ftft  IBM 


w  vv 

III. 

II 


y  •  ii  h 

i«  I., 


•  A 

!*.'*!** 

f  11 .  '?•" 

Ill  "  II 


SEPG 


f  ir*. 

•  . 11 T 

f  lb  n 

«  n 


©  2001  by  Carnegie  Mellon  University 


20 


Software  Erepneering  Institute 


Gather  Data 

Obtain  artifacts.  Process  documentation  is  identified  during  the 

interviews.  This  documentation  is  collected  immediately  or,  if 

necessary,  at  a  later  date. 

D 

D 

methods 

procedures 

0 

Q 

measurement 

templates 

| — ^ 

relevant 

metric-based 

policies 

documents  _ 

0 

process 

forms 

diagrams 

PCM  3:  Establish  TWG(s) 
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PCM  4:  Understand  Project's  Current 

State 

Project  Partner 
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Defining  the  existing 
process 
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PCM  5:  Redesign  the  Process 


New  Process  Design 
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New  Design  Emphasizes  Measurement 


The  new,  process-oriented  organization  puts  an  emphasis  on 
customers  and  the  value-added  processes  that  serve  them. 

Measurement  is  built  into  all  processes,  providing 
a  clear  view  of  the  current  situation  and  how  the  process 
performing. 

Measurement  enables  you  to  assess  proposed  changes 
and  the  results  of  change. 


Software  Engineering  Institute 


Why  the  E  mphasi  s  on  M  etri  cs? 

“Software  Engineering”  is  the  term  used  to  describe  the  collection 
of  techniques  concerned  with  applying  an  engineering  approach 
to  the  construction  of  software  products.  By  engineering 
approach  we  mean 


planning 

•  designing 

costing 

•  implementing 

managing 

•  testing 

modeling 

•  maintaining 

analyzing 

It  would  be  difficult  to  imagine  how  the  disciplines  of  electrical, 
mechanical  and  civil  engineering  could  have  evolved  without  a 
central  role  for  measurement. 
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Process  and  Product  Metrics 


Software  metrics  are  often  classified  as  either  process  metrics  or 
product  metrics. 

Process  metrics  quantify  attributes  of  the  development  process  and 
of  the  development  environment.  For  example: 

•  resources  (such  as  level  of  effort) 

•  duration  of  the  design  phase 

Product  metrics  are  measures  of  the  software  product.  For 
example: 

•  size  of  the  product  (such  as  lines  of  code,  number  of  modules) 

•  complexity  (such  as  flow  of  control,  depth  of  nesting,  or 
recursion) 

•  number  of  defects 

In  general,  it  is  likely  that  a  product 
metric  is  influenced  by  the  process  used 
and  vice  versa. 
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I  neffecti ve  Approach  to  Qual  ity 


Historically,  many  organizations  have  approached  quality  by 
performing  product  measurement  only.  But,  this  approach  does  not 
improve  quality  nor  guarantee  quality.  It’s  a  strategy  of  defect 
detection,  not  defect  prevention. 


Measurement  of  final  product  attributes  is  too 
late. ..the  quality,  good  or  bad,  is  already  in  the 
product. 

100%  inspection  is  only  80%  effective 

—  Dr.  Joseph  Juran 
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Traditional  Approach  is  Wasteful 


Input  Process  Rework  Acceptable 

product 
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Focus  on  the  Process 


Measurement  used 
to  understand 
and  improve  the 
process. 
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Input 
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o 
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T 


Process 


Treating  the  cause  rather 
than  the  symptoms. 


Acceptable 

product 


Rework 
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Business  Goals  and  Measures  Drive 
Project  Performance  Measures 
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Goal-Driven  Measurement 


The  Goal/Question/Metric  (GQM)  paradigm  provides  a  framework 
involving  three  steps. 


List  a  major  goal  of  the  process 

For  each  goal,  derive  questions  that  must  be 
answered  to  determine  if  the  goals  are  being 
met. 

3  )  Decide  what  must  be  measured  in  order  to  be 
able  to  answer  the  questions  adequately 


[Basili  88] 
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Software  M  etri  cs:  Ten  T raps  to  Avoi  d 

1  Lack  of  management  commitment 

2  Measuring  too  much,  too  soon 

3  Measuring  too  little,  too  late 

4  Measuring  the  wrong  things 

5  Imprecise  metrics  definitions 

6  Using  metrics  data  to  evaluate  individuals 

7  Using  metrics  to  motivate,  rather  than  to  understand 

8  Collecting  data  that  is  not  used 

9  Lack  of  communication  and  training 
Misinterpreting  metrics  data 


[Wiegers  00] 


©  2001  by  Carnegie  Mellon  University 


34 


SEPG 


CarnyMatat 

Software  Engineering  listitule 


PCM  6:  Develop  Solution 


Whole  product 
solution  .  •  • 


New  Process  Design 
from  PCM  5 


Guidebook 

for  users  of  the  process 
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PCM  7:  Conduct  Pilots  and  Evaluate 
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How  Will  You  Know  if  the  Change 
Worked? 

Quite  often,  pilot  studies  are  conducted  without  measurement 
either  before  or  after  the  change  was  introduced. 

Therefore,  how  do  we  know  if  the  outcome  was  better  or  worse 
than  the  original  situation? 

In  these  cases,  interpretation  is  based  on  opinion  and  impressions 
-  but  there  is  a  lack  of  data  to  back  it  up! 

Therefore,  there  is  less  confidence  in  the  results  of  the  pilot  - 
interpretation  is  problematic  and  there’s  a  risk  that  consensus 
about  the  results  are  not  achieved. 
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Typical  Approaches  to  Pi  lot  Study 
Evaluation 


Typical  approaches  that  fail 

Before  the  change  Change  introduced  After  the  Change 

Approach  #1  X 

Approach  #2  X  O 

Approach  #3  0  X 


represents  the  introduction  of  a  change 
O  represents  a  measurable  observation 
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“Engineering"  I  mplies  a  Scientific 
Approach 


Typical  approaches  that  fail 
Before  the  change 

Approach  #1 
Approach  #2 

Approach  #3  O 


Change  introduced 

X 


After  the  Change 


In  all  three  approaches,  there  is  no  way  to  tell  if  the  outcome  from  the 
change  was  better  or  worse  than  the  original  situation. 


How  do  you  know 
the  change  worked? 
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Approaches  to  Validation 

In  the  scientific  and  manufacturing  world,  improvements  or  innovations 
are  validated  using  a  rigorous  statistical  approach  known  as  design  of 
experiments  (DOE) 

•  Extraneous  variables  that  might  impact  the  result  you’re  looking  at 
can  be  held  steadied  or  controlled 

•  The  experimental  design  can  employ  techniques  such  as 
randomization  and  replication  to  add  clarity  and  confidence  to  the 
assertions  that  are  made  about  the  change 


Run 

Variable 

Variable 

Variable 

Number 

A 

B 

C 

1 

- 

- 

- 

2 

+ 

- 

+ 

3 

- 

+ 

- 

4 

+ 

+ 

+ 

Result 
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_  Software  Engineering  hstitute 

Scientific  Methods  Do  Exist 

Research  designs  called  quasi-experimentation  do  exist  for  proper 
interpretation  of  results  from  pilot  studies  ...  but  they  are  rarely  applied! 

a<3$9  Before  the  change 

Change  introduced 

After  the  Change 

aO 

&° 

X 

02 

r\V* 

Ay' 

Before  the  change 

Group  #1  0! 

Change  introduced 

X 

After  the  Change 

02 

Group  #2  03 

04 
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Consider  Better  Approach  #2 


Before  the  change 

Change  introduced 

After  the  Change 

Group  #1 

O, 

X 

02 

Group  #2 

03 

04 

Do  two  different  groups  obtain  the 
same  results  to  begin  with? 


If  yes,  then  02  and  04 
data  may  be  compared 
using  a  f  test. 


If  no,  then  02  and  04  data 
may  be  compared  using 

analysis  of  covariance. 


Note:  A  nest  and  the  analysis  of  covariance  method  are  statistical  methods  that  provide  a 
scientific  basis  for  making  assertions  about  the  results  of  your  pilot  study. 
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Software  Engineering  Institute _ 

F aci I itate  Organizati onal  Learni ng 

© — © — © — © — © — © — © — © 

Select  target  project(s)  and  plan  j 
Tailor  process  assets 
Support  and  monitor  project(s) 

Conduct  ongoing  evaluations  and  identify  new  improvement  opportunities 

Conduct  improvement  activities 
Communicate  results  and  update  the  process  asset  library 
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Conduct  "Lessons  Learned"  Events 


The  SEPG  should  conduct  regular  lessons  learned  events  after  each 
major  milestone  of  the  PCM. 

Many  organizations  conducting  PI  also  implement  yearly  “lessons 
learned”  events  to  “check  the  pulse”  of  the  organization. 

•  What  are  people’s  attitudes  about  the  changes  now  that  they’ve 
been  changed ? 


Now  that  some  are  experienced 
with  the  new  way  of  doing  things, 
what  ideas  are  there  for  improving 
what’s  there? 


Is  it 

working? 
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Assessing  PI  Progress 


The  SEPG  evaluates  the  progress  of  CMM  based  PI. 

The  Interim  Profile  method  is  a  way  to  rapidly  measure  an 
organization’s  SPI  maturity  between  assessments,  such  as 
a  CMM-Based  Appraisal  for  Internal  Process  Improvement  (CBA  IPI). 


[Whitney  96] 
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The  Process  Change  Method 


©  Organize  and  Prepare  ◄ - 

©  Conduct  Organizational  Scan 
®  Establish  Technical  Working  Groups 
©  Understand  Project’s  Current  State  < 

©  Redesign  the  Process  ◄ - 

©  Develop  Solutions - 


©  Conduct  Pilot(s)  and  Evaluate 
®  Facilitate  Organizational  Learning 
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